2026-10-29 20:17:25,365 [ 326180 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2026-10-29 20:17:25,365 [ 326180 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2026-10-29 20:17:25,365 [ 326180 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2026-10-29 20:17:25,365 [ 326180 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_iwvsmg --privileged --dns-search='.' --memory=30703247360 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel2_0.jsonl --report-log-exclude-logs-on-passed-tests 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1]' test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards test_part_uuid/test.py::test_part_uuid test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple test_partition/test.py::test_system_detached_parts test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_relative_filepath/test.py::test_filepath test_reload_client_certificate/test.py::test_correct_cn_cert test_reload_client_certificate/test.py::test_wrong_cn_cert test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_s3_imds/test_session_token.py::test_credentials_from_metadata test_s3_imds/test_simple.py::test_credentials_from_metadata test_s3_latency_log/test.py::test_latency_log test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 test_s3_storage_class/test.py::test_s3_storage_class_right 'test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3]' test_s3_style_link/test.py::test_s3_table_functions test_s3_style_link/test.py::test_s3_table_functions_line_as_string test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_recovery_replica/test.py::test_choose_source_replica test_partition/test.py::test_attach_check_all_parts test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] test_rocksdb_options/test.py::test_invalid_block_based_table_options test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_reloading_storage_configuration/test.py::test_add_disk test_replication_credentials/test.py::test_credentials_and_no_credentials [gw5] [ 1%] PASSED test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part [gw9] [ 2%] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 [gw5] [ 3%] PASSED test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists [gw0] [ 4%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw9] [ 5%] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 [gw2] [ 6%] PASSED test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options [gw9] [ 7%] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 [gw0] [ 8%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] [gw7] [ 9%] PASSED test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials [gw6] [ 10%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys [gw5] [ 11%] PASSED test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts [gw4] [ 12%] PASSED test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy [gw0] [ 13%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw6] [ 14%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces [gw5] [ 15%] PASSED test_partition/test.py::test_drop_detached_parts [gw7] [ 16%] PASSED test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_partition/test.py::test_make_clone_in_detached [gw0] [ 17%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] [gw0] [ 18%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] [gw7] [ 19%] PASSED test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials [gw4] [ 20%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config [gw6] [ 21%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys [gw5] [ 22%] PASSED test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex [gw0] [ 23%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] [gw2] [ 24%] PASSED test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options [gw3] [ 25%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] [gw0] [ 26%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw7] [ 27%] PASSED test_replication_credentials/test.py::test_same_credentials [gw5] [ 28%] PASSED test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple [gw3] [ 29%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] [gw0] [ 30%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] [gw5] [ 31%] PASSED test_partition/test.py::test_partition_simple [gw6] [ 32%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_same_keys test_partition/test.py::test_system_detached_parts [gw4] [ 33%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards [gw8] [ 34%] PASSED test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery [gw0] [ 35%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] [gw3] [ 36%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] test_reload_client_certificate/test.py::test_correct_cn_cert test_replicated_table_attach/test.py::test_startup_with_small_bg_pool [gw2] [ 37%] PASSED test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options [gw0] [ 38%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] [gw4] [ 39%] PASSED test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy [gw3] [ 40%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] [gw0] [ 41%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw8] [ 42%] PASSED test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata [gw1] [ 43%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] [gw0] [ 44%] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] [gw4] [ 45%] PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works [gw3] [ 46%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] [gw1] [ 47%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] test_range_hashed_dictionary_types/test.py::test_range_hashed_dict [gw5] [ 48%] PASSED test_partition/test.py::test_system_detached_parts [gw2] [ 49%] PASSED test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options [gw1] [ 50%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] [gw3] [ 51%] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] [gw4] [ 52%] PASSED test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_s3_style_link/test.py::test_s3_table_functions [gw7] [ 53%] PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned [gw1] [ 54%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] [gw1] [ 55%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] [gw4] [ 56%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy [gw0] [ 57%] PASSED test_range_hashed_dictionary_types/test.py::test_range_hashed_dict [gw9] [ 58%] PASSED test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards test_read_only_table/test.py::test_restart_zookeeper [gw2] [ 59%] PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options [gw5] [ 60%] PASSED test_s3_style_link/test.py::test_s3_table_functions test_s3_style_link/test.py::test_s3_table_functions_line_as_string [gw1] [ 61%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] test_relative_filepath/test.py::test_filepath [gw5] [ 62%] PASSED test_s3_style_link/test.py::test_s3_table_functions_line_as_string [gw4] [ 63%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy [gw1] [ 64%] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge [gw8] [ 65%] PASSED test_recovery_replica/test.py::test_update_metadata [gw6] [ 66%] PASSED test_reload_client_certificate/test.py::test_correct_cn_cert test_reload_client_certificate/test.py::test_wrong_cn_cert test_parallel_replicas_increase_error_count/test.py::test_increase_error_count [gw4] [ 67%] PASSED test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy [gw0] [ 68%] PASSED test_relative_filepath/test.py::test_filepath [gw1] [ 69%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter [gw2] [ 70%] PASSED test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options [gw2] [ 71%] PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options [gw2] [ 72%] PASSED test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach [gw2] [ 73%] PASSED test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options [gw2] [ 74%] PASSED test_rocksdb_options/test.py::test_valid_options [gw4] [ 75%] PASSED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replicating_constants/test.py::test_different_versions [gw7] [ 76%] PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned [gw4] [ 77%] PASSED test_reloading_storage_configuration/test.py::test_update_disk test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race [gw0] [ 78%] PASSED test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach [gw1] [ 79%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system [gw9] [ 80%] PASSED test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards [gw8] [ 81%] PASSED test_parallel_replicas_increase_error_count/test.py::test_increase_error_count [gw5] [ 82%] PASSED test_replicating_constants/test.py::test_different_versions test_s3_latency_log/test.py::test_latency_log test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_s3_table_functions/test.py::test_s3_table_functions [gw3] [ 83%] PASSED test_read_only_table/test.py::test_restart_zookeeper test_s3_storage_class/test.py::test_s3_storage_class_right [gw1] [ 84%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation [gw7] [ 85%] FAILED test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_s3_imds/test_session_token.py::test_credentials_from_metadata [gw0] [ 86%] PASSED test_s3_storage_class/test.py::test_s3_storage_class_right [gw9] [ 87%] PASSED test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts [gw8] [ 88%] PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop [gw3] [ 89%] PASSED test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_s3_imds/test_simple.py::test_credentials_from_metadata [gw7] [ 90%] PASSED test_s3_imds/test_session_token.py::test_credentials_from_metadata [gw9] [ 91%] PASSED test_s3_table_functions/test.py::test_s3_table_functions_timeouts [gw5] [ 92%] PASSED test_s3_latency_log/test.py::test_latency_log [gw1] [ 93%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter [gw3] [ 94%] PASSED test_s3_imds/test_simple.py::test_credentials_from_metadata [gw1] [ 95%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system [gw1] [ 96%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] [gw1] [ 97%] PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] [gw6] [ 98%] PASSED test_reload_client_certificate/test.py::test_wrong_cn_cert test_part_uuid/test.py::test_part_uuid [gw6] [ 99%] PASSED test_part_uuid/test.py::test_part_uuid test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] [gw6] [100%] PASSED test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] =================================== FAILURES =================================== ________________________ test_partition_move_drop_race _________________________ [gw7] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_partition_move_drop_race(started_cluster): for node in [node1, node2, node3]: node.query(f"DROP DATABASE IF EXISTS test_db SYNC") node1.query("SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range") node1.query("SYSTEM ENABLE FAILPOINT rmt_delay_commit_part") for node in [node1, node2, node3]: node.query( """ CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') """ ) node1.query( f""" CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 """ ) for node in [node1, node2, node3]: assert node.query_with_retry( "SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl'", check_callback=lambda x: x.strip() == "1", ) table_uuid = node1.query( "SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl'" ).strip() node2.query(f"SYSTEM STOP FETCHES test_db.tbl") node3.query(f"SYSTEM STOP FETCHES test_db.tbl") prev_block_number = get_last_block_number_for_partition(node1, table_uuid, 0) exception_holder = [None] def drop_op(): try: # Wait until INSERT query allocates the block number timeout = 60 # seconds start_time = time.time() while True: current_block_number = get_last_block_number_for_partition( node1, table_uuid, 0 ) if current_block_number != prev_block_number: break if time.time() - start_time > timeout: raise TimeoutError( f"Timeout waiting for block number to change after {timeout}s. " f"Previous: {prev_block_number}, Current: {current_block_number}" ) time.sleep(0.5) node1.query(f"ALTER TABLE test_db.tbl DROP PARTITION 0") except Exception as e: exception_holder[0] = e t = threading.Thread(target=drop_op) t.start() > node1.query(f"INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c')") test_race_condition_for_replicated_merge_tree/test.py:146: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3649: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 138, stderr: Received exception from server (version 25.3.14): E Code: 394. DB::Exception: Received from 172.16.4.7:9000. DB::Exception: Insert query (for block /clickhouse/tables/fae174c5-137a-4afb-bc42-f4e8de019cc5/s1/block_numbers/0/block-0000000000) was canceled by concurrent ALTER PARTITION or TRUNCATE. Stack trace: E E 0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::Exception::Exception(String const&, int) @ 0x00000000210ee800 E 1. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000010dfe294 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000008ab0700 E 3. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x0000000008ab0300 E 4. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1116: DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda1'()::operator()() const @ 0x000000001ba791af E 5. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1146: DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda2'()::operator()() const @ 0x000000001ba768b9 E 6. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1179: void DB::ZooKeeperRetriesControl::retryLoop::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&, void DB::ZooKeeperRetriesControl::retryLoop::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()>(DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&&)::'lambda'()>(DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&&, void DB::ZooKeeperRetriesControl::retryLoop::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()>(DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long)::'lambda3'()&&)::'lambda'()&&) @ 0x000000001ba76691 E 7. ./src/Common/ZooKeeper/ZooKeeperRetries.h:52: DB::ReplicatedMergeTreeSinkImpl::commitPart(std::shared_ptr const&, std::shared_ptr&, String const&, unsigned long) @ 0x000000001ba5fc6b E 8. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:493: DB::ReplicatedMergeTreeSinkImpl::finishDelayedChunk(std::shared_ptr const&) @ 0x000000001ba4ccd8 E 9. ./build_docker/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:1257: DB::ReplicatedMergeTreeSinkImpl::onFinish() @ 0x000000001ba5f02f E 10. ./build_docker/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:187: void std::__function::__policy_invoker::__call_impl[abi:ne190107]>(std::__function::__policy_storage const*) @ 0x000000001c3b4951 E 11. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001c3b43b7 E 12. ./build_docker/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:187: DB::ExceptionKeepingTransform::work() @ 0x000000001c3b3dc1 E 13. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::ExecutionThreadContext::executeTask() @ 0x000000001c021786 E 14. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:296: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x000000001c010ffb E 15. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:170: DB::PipelineExecutor::executeStep(std::atomic*) @ 0x000000001c010835 E 16. ./build_docker/./src/Processors/Executors/PushingPipelineExecutor.cpp:131: DB::PushingPipelineExecutor::finish() @ 0x000000001c02b4ce E 17. ./build_docker/./src/Server/TCPHandler.cpp:1128: DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x000000001beea8a6 E 18. ./build_docker/./src/Server/TCPHandler.cpp:673: DB::TCPHandler::runImpl() @ 0x000000001bed9ff5 E 19. ./build_docker/./src/Server/TCPHandler.cpp:2629: DB::TCPHandler::run() @ 0x000000001bf00c88 E 20. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:40: Poco::Net::TCPServerConnection::start() @ 0x00000000211fc243 E 21. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x00000000211fcab2 E 22. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:205: Poco::PooledThread::run() @ 0x0000000021176e03 E 23. ./build_docker/./base/poco/Foundation/src/Thread.cpp:45: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x00000000211751b0 E 24. ./base/poco/Foundation/src/Thread_POSIX.cpp:335: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000002117356a E 25. __tsan_thread_start_func @ 0x0000000008a23e68 E 26. ? @ 0x00007f8014940ac3 E 27. ? @ 0x00007f80149d2850 E . (QUERY_WAS_CANCELLED) E (query: INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c')) helpers/client.py:248: QueryRuntimeException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- Running tests in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/test.py Running tests in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/test.py Cluster start called. is_up=False Cluster start called. is_up=False Docker networks for project roottestraceconditionforreplicatedmergetree-gw7 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestraceconditionforreplicatedmergetree-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestraceconditionforreplicatedmergetree-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestraceconditionforreplicatedmergetree-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestraceconditionforreplicatedmergetree-gw7 are DRIVER VOLUME NAME Docker volumes for project roottestraceconditionforreplicatedmergetree-gw7 are DRIVER VOLUME NAME Cleanup called Cleanup called Docker networks for project roottestraceconditionforreplicatedmergetree-gw7 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestraceconditionforreplicatedmergetree-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestraceconditionforreplicatedmergetree-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestraceconditionforreplicatedmergetree-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestraceconditionforreplicatedmergetree-gw7 are DRIVER VOLUME NAME Docker volumes for project roottestraceconditionforreplicatedmergetree-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestraceconditionforreplicatedmergetree-gw7 No running containers for project: roottestraceconditionforreplicatedmergetree-gw7 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:6 Stdout:6 Command:[docker volume prune -f] Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Volumes pruned: 6 Volumes pruned: 6 Setup directory for instance: node1 Setup directory for instance: node1 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/database Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/database Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/logs Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node2 Setup directory for instance: node2 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/configs/config.d Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/database Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/database Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/logs Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Setup directory for instance: node3 Setup directory for instance: node3 Create directory for configuration generated in this helper Create directory for configuration generated in this helper Create directory for common tests configuration Create directory for common tests configuration Copy common configuration from helpers Copy common configuration from helpers Generate and write macros file Generate and write macros file Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/configs/config.d Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/configs/config.d Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/database Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/database Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/logs Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/logs Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml pull] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml pull] Stderr: zoo1 Skipped - Image is already being pulled by zoo3 Stderr: zoo1 Skipped - Image is already being pulled by zoo3 Stderr: node2 Skipped - Image is already being pulled by zoo3 Stderr: node2 Skipped - Image is already being pulled by zoo3 Stderr: node3 Skipped - Image is already being pulled by zoo3 Stderr: node3 Skipped - Image is already being pulled by zoo3 Stderr: node1 Skipped - Image is already being pulled by zoo3 Stderr: node1 Skipped - Image is already being pulled by zoo3 Stderr: zoo2 Skipped - Image is already being pulled by zoo3 Stderr: zoo2 Skipped - Image is already being pulled by zoo3 Stderr: zoo3 Pulling Stderr: zoo3 Pulling Stderr: zoo3 Pulled Stderr: zoo3 Pulled Setup ZooKeeper Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/coordination'] Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/coordination'] Command:[docker compose --project-name roottestraceconditionforreplicatedmergetree-gw7 --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Command:[docker compose --project-name roottestraceconditionforreplicatedmergetree-gw7 --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr:time="2026-10-29T20:19:38Z" level=trace msg="Docker Desktop integration not enabled" Stderr:time="2026-10-29T20:19:38Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Creating Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Creating Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Created Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Started Stderr:time="2026-10-29T20:19:39Z" level=debug msg="otel error" error="" Stderr:time="2026-10-29T20:19:39Z" level=debug msg="otel error" error="" Stderr:time="2026-10-29T20:19:39Z" level=debug msg="otel error" error="" Stderr:time="2026-10-29T20:19:39Z" level=debug msg="otel error" error="" Wait ZooKeeper to start Wait ZooKeeper to start get_instance_ip instance_name=zoo1 get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-zoo1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.4.2, port:2181, use_ssl:False get_kazoo_client: zoo1, ip:172.16.4.2, port:2181, use_ssl:False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connection dropped: socket connection error: Connection refused Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-zoo2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.4.4, port:2181, use_ssl:False get_kazoo_client: zoo2, ip:172.16.4.4, port:2181, use_ssl:False Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-zoo3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.4.3, port:2181, use_ssl:False get_kazoo_client: zoo3, ip:172.16.4.3, port:2181, use_ssl:False Connecting to 172.16.4.3(172.16.4.3):2181, use_ssl: False Connecting to 172.16.4.3(172.16.4.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Sending request(xid=2): Close() Connection dropped: socket connection broken Connection dropped: socket connection broken Transition to CONNECTING Transition to CONNECTING Zookeeper connection lost Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml up -d --no-recreate') ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml up -d --no-recreate] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml up -d --no-recreate] Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Running Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Creating Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Created Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Starting Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Started Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Started ClickHouse instance created ClickHouse instance created get_instance_ip instance_name=node1 get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node1 get_instance_ip instance_name=node1 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node1, ip: 172.16.4.7... Waiting for ClickHouse start in node1, ip: 172.16.4.7... http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None ClickHouse node1 started ClickHouse node1 started get_instance_ip instance_name=node2 get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node2-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node2 get_instance_ip instance_name=node2 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node2-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node2, ip: 172.16.4.6... Waiting for ClickHouse start in node2, ip: 172.16.4.6... http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node2-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/793e06dacb2beffe8051fc639fb0de8736a917bc9e8d235b098d1621543d568f/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/793e06dacb2beffe8051fc639fb0de8736a917bc9e8d235b098d1621543d568f/json HTTP/1.1" 200 None ClickHouse node2 started ClickHouse node2 started get_instance_ip instance_name=node3 get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node3-1/json HTTP/1.1" 200 None get_instance_ip instance_name=node3 get_instance_ip instance_name=node3 http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node3-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in node3, ip: 172.16.4.5... Waiting for ClickHouse start in node3, ip: 172.16.4.5... http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node3-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c93c6ae633fe61e3856166e39d7903fbe9fb0a6b5cf5c9b37f907a7fbd95d424/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/c93c6ae633fe61e3856166e39d7903fbe9fb0a6b5cf5c9b37f907a7fbd95d424/json HTTP/1.1" 200 None ClickHouse node3 started ClickHouse node3 started ------------------------------ Captured log setup ------------------------------ 2026-10-29 20:19:36.508000 [ 695 ] INFO : Running tests in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/test.py (cluster.py:2738, start) 2026-10-29 20:19:36.508000 [ 695 ] DEBUG : Cluster start called. is_up=False (cluster.py:2745, start) 2026-10-29 20:19:36.526000 [ 695 ] DEBUG : Docker networks for project roottestraceconditionforreplicatedmergetree-gw7 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 20:19:36.543000 [ 695 ] DEBUG : Docker containers for project roottestraceconditionforreplicatedmergetree-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 20:19:36.569000 [ 695 ] DEBUG : Docker volumes for project roottestraceconditionforreplicatedmergetree-gw7 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 20:19:36.569000 [ 695 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 20:19:36.586000 [ 695 ] DEBUG : Docker networks for project roottestraceconditionforreplicatedmergetree-gw7 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 20:19:36.604000 [ 695 ] DEBUG : Docker containers for project roottestraceconditionforreplicatedmergetree-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 20:19:36.623000 [ 695 ] DEBUG : Docker volumes for project roottestraceconditionforreplicatedmergetree-gw7 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 20:19:36.623000 [ 695 ] DEBUG : Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 20:19:36.643000 [ 695 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 20:19:36.643000 [ 695 ] DEBUG : No running containers for project: roottestraceconditionforreplicatedmergetree-gw7 (cluster.py:879, cleanup) 2026-10-29 20:19:36.643000 [ 695 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 20:19:36.664000 [ 695 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 20:19:36.664000 [ 695 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 20:19:36.689000 [ 695 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 20:19:36.689000 [ 695 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 20:19:36.689000 [ 695 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 20:19:36.689000 [ 695 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 20:19:36.710000 [ 695 ] DEBUG : Stdout:6 (cluster.py:145, run_and_check) 2026-10-29 20:19:36.710000 [ 695 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-10-29 20:19:36.730000 [ 695 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 20:19:36.731000 [ 695 ] DEBUG : Volumes pruned: 6 (cluster.py:915, cleanup) 2026-10-29 20:19:36.731000 [ 695 ] DEBUG : Setup directory for instance: node1 (cluster.py:2758, start) 2026-10-29 20:19:36.731000 [ 695 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 20:19:36.731000 [ 695 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 20:19:36.732000 [ 695 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 20:19:36.732000 [ 695 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 20:19:36.732000 [ 695 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 20:19:36.733000 [ 695 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/database (cluster.py:4758, create_dir) 2026-10-29 20:19:36.733000 [ 695 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/logs (cluster.py:4769, create_dir) 2026-10-29 20:19:36.733000 [ 695 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4850, create_dir) 2026-10-29 20:19:36.733000 [ 695 ] DEBUG : Setup directory for instance: node2 (cluster.py:2758, start) 2026-10-29 20:19:36.734000 [ 695 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 20:19:36.734000 [ 695 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 20:19:36.734000 [ 695 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 20:19:36.734000 [ 695 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 20:19:36.735000 [ 695 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 20:19:36.735000 [ 695 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/database (cluster.py:4758, create_dir) 2026-10-29 20:19:36.735000 [ 695 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/logs (cluster.py:4769, create_dir) 2026-10-29 20:19:36.735000 [ 695 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4850, create_dir) 2026-10-29 20:19:36.735000 [ 695 ] DEBUG : Setup directory for instance: node3 (cluster.py:2758, start) 2026-10-29 20:19:36.735000 [ 695 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-10-29 20:19:36.735000 [ 695 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-10-29 20:19:36.736000 [ 695 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-10-29 20:19:36.736000 [ 695 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-10-29 20:19:36.737000 [ 695 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/configs/config.d (cluster.py:4741, create_dir) 2026-10-29 20:19:36.737000 [ 695 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/database (cluster.py:4758, create_dir) 2026-10-29 20:19:36.737000 [ 695 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/logs (cluster.py:4769, create_dir) 2026-10-29 20:19:36.737000 [ 695 ] DEBUG : Entrypoint cmd: bash -c "trap 'pkill tail' INT TERM; clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --daemon -- ; coproc tail -f /dev/null; wait $$!" (cluster.py:4850, create_dir) 2026-10-29 20:19:36.737000 [ 695 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env (cluster.py:96, _create_env_file) 2026-10-29 20:19:36.737000 [ 695 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 20:19:36.737000 [ 695 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 20:19:36.737000 [ 695 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-10-29 20:19:36.738000 [ 695 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-10-29 20:19:36.746000 [ 695 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2026-10-29 20:19:36.746000 [ 695 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-10-29 20:19:38.750000 [ 695 ] DEBUG : Stderr: zoo1 Skipped - Image is already being pulled by zoo3 (cluster.py:147, run_and_check) 2026-10-29 20:19:38.750000 [ 695 ] DEBUG : Stderr: node2 Skipped - Image is already being pulled by zoo3 (cluster.py:147, run_and_check) 2026-10-29 20:19:38.750000 [ 695 ] DEBUG : Stderr: node3 Skipped - Image is already being pulled by zoo3 (cluster.py:147, run_and_check) 2026-10-29 20:19:38.750000 [ 695 ] DEBUG : Stderr: node1 Skipped - Image is already being pulled by zoo3 (cluster.py:147, run_and_check) 2026-10-29 20:19:38.750000 [ 695 ] DEBUG : Stderr: zoo2 Skipped - Image is already being pulled by zoo3 (cluster.py:147, run_and_check) 2026-10-29 20:19:38.750000 [ 695 ] DEBUG : Stderr: zoo3 Pulling (cluster.py:147, run_and_check) 2026-10-29 20:19:38.750000 [ 695 ] DEBUG : Stderr: zoo3 Pulled (cluster.py:147, run_and_check) 2026-10-29 20:19:38.750000 [ 695 ] DEBUG : Setup ZooKeeper (cluster.py:2799, start) 2026-10-29 20:19:38.750000 [ 695 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper1/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper2/coordination', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/log', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/config', '/ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/keeper3/coordination'] (cluster.py:2800, start) 2026-10-29 20:19:38.751000 [ 695 ] DEBUG : Command:[docker compose --project-name roottestraceconditionforreplicatedmergetree-gw7 --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] (cluster.py:121, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr:time="2026-10-29T20:19:38Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Creating (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Created (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:19:39.878000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:19:39.879000 [ 695 ] DEBUG : Stderr:time="2026-10-29T20:19:39Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-10-29 20:19:39.879000 [ 695 ] DEBUG : Stderr:time="2026-10-29T20:19:39Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2026-10-29 20:19:39.879000 [ 695 ] DEBUG : Wait ZooKeeper to start (cluster.py:2436, wait_zookeeper_to_start) 2026-10-29 20:19:39.879000 [ 695 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:2005, get_instance_ip) 2026-10-29 20:19:39.880000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-zoo1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:39.881000 [ 695 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.4.2, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-10-29 20:19:39.882000 [ 695 ] INFO : Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:19:39.882000 [ 695 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 20:19:40.004000 [ 695 ] INFO : Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:19:40.004000 [ 695 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 20:19:40.289000 [ 695 ] INFO : Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:19:40.290000 [ 695 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 20:19:41.025000 [ 695 ] INFO : Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:19:41.025000 [ 695 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 20:19:42.871000 [ 695 ] INFO : Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:19:42.872000 [ 695 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2026-10-29 20:19:46.707000 [ 695 ] INFO : Connecting to 172.16.4.2(172.16.4.2):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:19:46.708000 [ 695 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-10-29 20:19:46.714000 [ 695 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-10-29 20:19:46.715000 [ 695 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-10-29 20:19:46.716000 [ 695 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-10-29 20:19:46.716000 [ 695 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-10-29 20:19:46.720000 [ 695 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-10-29 20:19:46.721000 [ 695 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-10-29 20:19:46.721000 [ 695 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-10-29 20:19:46.798000 [ 695 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-10-29 20:19:46.799000 [ 695 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-10-29 20:19:46.799000 [ 695 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:2005, get_instance_ip) 2026-10-29 20:19:46.801000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-zoo2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:46.801000 [ 695 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.4.4, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-10-29 20:19:46.802000 [ 695 ] INFO : Connecting to 172.16.4.4(172.16.4.4):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:19:46.802000 [ 695 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-10-29 20:19:46.805000 [ 695 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-10-29 20:19:46.806000 [ 695 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-10-29 20:19:46.806000 [ 695 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-10-29 20:19:46.806000 [ 695 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-10-29 20:19:46.809000 [ 695 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-10-29 20:19:46.809000 [ 695 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-10-29 20:19:46.809000 [ 695 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-10-29 20:19:46.871000 [ 695 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-10-29 20:19:46.871000 [ 695 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-10-29 20:19:46.871000 [ 695 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:2005, get_instance_ip) 2026-10-29 20:19:46.873000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-zoo3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:46.874000 [ 695 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.4.3, port:2181, use_ssl:False (cluster.py:3312, get_kazoo_client) 2026-10-29 20:19:46.875000 [ 695 ] INFO : Connecting to 172.16.4.3(172.16.4.3):2181, use_ssl: False (connection.py:650, _connect) 2026-10-29 20:19:46.875000 [ 695 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2026-10-29 20:19:46.881000 [ 695 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2026-10-29 20:19:46.882000 [ 695 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2026-10-29 20:19:46.884000 [ 695 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2026-10-29 20:19:46.884000 [ 695 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2026-10-29 20:19:46.888000 [ 695 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2026-10-29 20:19:46.888000 [ 695 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2026-10-29 20:19:46.888000 [ 695 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2026-10-29 20:19:46.988000 [ 695 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2026-10-29 20:19:46.989000 [ 695 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2026-10-29 20:19:46.989000 [ 695 ] DEBUG : All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') (cluster.py:2452, wait_zookeeper_nodes_to_start) 2026-10-29 20:19:46.989000 [ 695 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml up -d --no-recreate') (cluster.py:3139, start) 2026-10-29 20:19:46.989000 [ 695 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml up -d --no-recreate] (cluster.py:121, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Running (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Running (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Running (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Creating (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Created (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Starting (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Started (cluster.py:147, run_and_check) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : ClickHouse instance created (cluster.py:3147, start) 2026-10-29 20:19:47.759000 [ 695 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2005, get_instance_ip) 2026-10-29 20:19:47.761000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:47.762000 [ 695 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 20:19:47.763000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:47.763000 [ 695 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.4.7... (cluster.py:3155, start) 2026-10-29 20:19:47.764000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:47.766000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:47.868000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:47.971000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:48.073000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:48.175000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:48.281000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:48.384000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:48.487000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:48.589000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:48.691000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:48.794000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:48.896000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:48.999000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:49.102000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:49.205000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:49.307000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:49.410000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:49.513000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:49.615000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:49.717000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:49.820000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:49.923000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.025000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.128000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/4599e96887714e0afd300812aac7c7efd41a79f13b25a7a263992d506b12ed18/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.128000 [ 695 ] DEBUG : ClickHouse node1 started (cluster.py:3159, start) 2026-10-29 20:19:50.128000 [ 695 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2005, get_instance_ip) 2026-10-29 20:19:50.130000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.130000 [ 695 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 20:19:50.131000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.131000 [ 695 ] DEBUG : Waiting for ClickHouse start in node2, ip: 172.16.4.6... (cluster.py:3155, start) 2026-10-29 20:19:50.132000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.134000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/793e06dacb2beffe8051fc639fb0de8736a917bc9e8d235b098d1621543d568f/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.134000 [ 695 ] DEBUG : ClickHouse node2 started (cluster.py:3159, start) 2026-10-29 20:19:50.134000 [ 695 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2005, get_instance_ip) 2026-10-29 20:19:50.135000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.135000 [ 695 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2015, get_instance_global_ipv6) 2026-10-29 20:19:50.136000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.137000 [ 695 ] DEBUG : Waiting for ClickHouse start in node3, ip: 172.16.4.5... (cluster.py:3155, start) 2026-10-29 20:19:50.137000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottestraceconditionforreplicatedmergetree-gw7-node3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.139000 [ 695 ] DEBUG : http://localhost:None "GET /v1.46/containers/c93c6ae633fe61e3856166e39d7903fbe9fb0a6b5cf5c9b37f907a7fbd95d424/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-10-29 20:19:50.139000 [ 695 ] DEBUG : ClickHouse node3 started (cluster.py:3159, start) ----------------------------- Captured stderr call ----------------------------- Executing query DROP DATABASE IF EXISTS test_db SYNC on node1 Executing query DROP DATABASE IF EXISTS test_db SYNC on node1 Executing query DROP DATABASE IF EXISTS test_db SYNC on node2 Executing query DROP DATABASE IF EXISTS test_db SYNC on node2 Executing query DROP DATABASE IF EXISTS test_db SYNC on node3 Executing query DROP DATABASE IF EXISTS test_db SYNC on node3 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range on node1 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range on node1 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_commit_part on node1 Executing query SYSTEM ENABLE FAILPOINT rmt_delay_commit_part on node1 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node1 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node1 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node2 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node2 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node3 Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node3 Executing query CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 on node1 Executing query CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 on node1 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node2 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node2 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node3 Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node3 Executing query SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl' on node1 Executing query SYSTEM STOP FETCHES test_db.tbl on node2 Executing query SYSTEM STOP FETCHES test_db.tbl on node2 Executing query SYSTEM STOP FETCHES test_db.tbl on node3 Executing query SYSTEM STOP FETCHES test_db.tbl on node3 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fae174c5-137a-4afb-bc42-f4e8de019cc5/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fae174c5-137a-4afb-bc42-f4e8de019cc5/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fae174c5-137a-4afb-bc42-f4e8de019cc5/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fae174c5-137a-4afb-bc42-f4e8de019cc5/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c') on node1 Executing query INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c') on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fae174c5-137a-4afb-bc42-f4e8de019cc5/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fae174c5-137a-4afb-bc42-f4e8de019cc5/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 Executing query ALTER TABLE test_db.tbl DROP PARTITION 0 on node1 Executing query ALTER TABLE test_db.tbl DROP PARTITION 0 on node1 ------------------------------ Captured log call ------------------------------- 2026-10-29 20:19:50.178000 [ 695 ] DEBUG : Executing query DROP DATABASE IF EXISTS test_db SYNC on node1 (cluster.py:3648, query) 2026-10-29 20:19:50.644000 [ 695 ] DEBUG : Executing query DROP DATABASE IF EXISTS test_db SYNC on node2 (cluster.py:3648, query) 2026-10-29 20:19:50.860000 [ 695 ] DEBUG : Executing query DROP DATABASE IF EXISTS test_db SYNC on node3 (cluster.py:3648, query) 2026-10-29 20:19:51.326000 [ 695 ] DEBUG : Executing query SYSTEM ENABLE FAILPOINT rmt_delay_execute_drop_range on node1 (cluster.py:3648, query) 2026-10-29 20:19:51.592000 [ 695 ] DEBUG : Executing query SYSTEM ENABLE FAILPOINT rmt_delay_commit_part on node1 (cluster.py:3648, query) 2026-10-29 20:19:51.807000 [ 695 ] DEBUG : Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node1 (cluster.py:3648, query) 2026-10-29 20:19:52.073000 [ 695 ] DEBUG : Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node2 (cluster.py:3648, query) 2026-10-29 20:19:52.488000 [ 695 ] DEBUG : Executing query CREATE DATABASE test_db ENGINE = Replicated('/test/db', '{shard}', '{replica}') on node3 (cluster.py:3648, query) 2026-10-29 20:19:52.754000 [ 695 ] DEBUG : Executing query CREATE TABLE test_db.tbl (id UInt32, val String) ENGINE = ReplicatedMergeTree PARTITION BY id % 10 ORDER BY id SETTINGS old_parts_lifetime = 1 on node1 (cluster.py:3648, query) 2026-10-29 20:19:53.169000 [ 695 ] DEBUG : Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node1 (cluster.py:3648, query) 2026-10-29 20:19:53.485000 [ 695 ] DEBUG : Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node2 (cluster.py:3648, query) 2026-10-29 20:19:53.801000 [ 695 ] DEBUG : Executing query SELECT count() FROM system.tables WHERE database='test_db' AND name='tbl' on node3 (cluster.py:3648, query) 2026-10-29 20:19:54.167000 [ 695 ] DEBUG : Executing query SELECT uuid FROM system.tables WHERE database='test_db' AND name='tbl' on node1 (cluster.py:3648, query) 2026-10-29 20:19:54.432000 [ 695 ] DEBUG : Executing query SYSTEM STOP FETCHES test_db.tbl on node2 (cluster.py:3648, query) 2026-10-29 20:19:54.647000 [ 695 ] DEBUG : Executing query SYSTEM STOP FETCHES test_db.tbl on node3 (cluster.py:3648, query) 2026-10-29 20:19:54.863000 [ 695 ] DEBUG : Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fae174c5-137a-4afb-bc42-f4e8de019cc5/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 (cluster.py:3648, query) 2026-10-29 20:19:55.178000 [ 695 ] DEBUG : Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fae174c5-137a-4afb-bc42-f4e8de019cc5/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 (cluster.py:3648, query) 2026-10-29 20:19:55.178000 [ 695 ] DEBUG : Executing query INSERT INTO test_db.tbl VALUES (0, 'a'), (10, 'b'), (20, 'c') on node1 (cluster.py:3648, query) 2026-10-29 20:19:55.945000 [ 695 ] DEBUG : Executing query SELECT toInt64(replaceRegexpOne(name, 'block-', '')) as block_number FROM system.zookeeper WHERE path = '/clickhouse/tables/fae174c5-137a-4afb-bc42-f4e8de019cc5/s1/block_numbers/0' AND name LIKE 'block-%' ORDER BY block_number DESC LIMIT 1 on node1 (cluster.py:3648, query) 2026-10-29 20:19:56.311000 [ 695 ] DEBUG : Executing query ALTER TABLE test_db.tbl DROP PARTITION 0 on node1 (cluster.py:3648, query) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml stop --timeout 20] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml stop --timeout 20] Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml down --volumes] Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml down --volumes] Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopping Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopped Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Removing Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Removed Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Removed Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Removing Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Removing Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Removed Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Removed Cleanup called Cleanup called Docker networks for project roottestraceconditionforreplicatedmergetree-gw7 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottestraceconditionforreplicatedmergetree-gw7 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestraceconditionforreplicatedmergetree-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottestraceconditionforreplicatedmergetree-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestraceconditionforreplicatedmergetree-gw7 are DRIVER VOLUME NAME Docker volumes for project roottestraceconditionforreplicatedmergetree-gw7 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottestraceconditionforreplicatedmergetree-gw7 No running containers for project: roottestraceconditionforreplicatedmergetree-gw7 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:12 Stdout:12 Command:[docker volume prune -f] Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Volumes pruned: 12 Volumes pruned: 12 ---------------------------- Captured log teardown ----------------------------- 2026-10-29 20:20:00.798000 [ 695 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml stop --timeout 20] (cluster.py:121, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.197000 [ 695 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 20:20:03.208000 [ 695 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 20:20:03.218000 [ 695 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2026-10-29 20:20:03.228000 [ 695 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/.env --project-name roottestraceconditionforreplicatedmergetree-gw7 --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node1/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_race_condition_for_replicated_merge_tree/_instances-0-gw7/node3/docker-compose.yml down --volumes] (cluster.py:121, run_and_check) 2026-10-29 20:20:03.995000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node2-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node3-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-node1-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopping (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Stopped (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Removing (cluster.py:147, run_and_check) 2026-10-29 20:20:03.996000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo3-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:20:03.997000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo2-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:20:03.997000 [ 695 ] DEBUG : Stderr: Container roottestraceconditionforreplicatedmergetree-gw7-zoo1-1 Removed (cluster.py:147, run_and_check) 2026-10-29 20:20:03.997000 [ 695 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Removing (cluster.py:147, run_and_check) 2026-10-29 20:20:03.997000 [ 695 ] DEBUG : Stderr: Network roottestraceconditionforreplicatedmergetree-gw7_default Removed (cluster.py:147, run_and_check) 2026-10-29 20:20:03.997000 [ 695 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-10-29 20:20:04.013000 [ 695 ] DEBUG : Docker networks for project roottestraceconditionforreplicatedmergetree-gw7 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-10-29 20:20:04.029000 [ 695 ] DEBUG : Docker containers for project roottestraceconditionforreplicatedmergetree-gw7 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-10-29 20:20:04.044000 [ 695 ] DEBUG : Docker volumes for project roottestraceconditionforreplicatedmergetree-gw7 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-10-29 20:20:04.044000 [ 695 ] DEBUG : Command:[docker container list --all --filter name='^/roottestraceconditionforreplicatedmergetree-gw7-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-10-29 20:20:04.059000 [ 695 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-10-29 20:20:04.059000 [ 695 ] DEBUG : No running containers for project: roottestraceconditionforreplicatedmergetree-gw7 (cluster.py:879, cleanup) 2026-10-29 20:20:04.059000 [ 695 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-10-29 20:20:04.075000 [ 695 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-10-29 20:20:04.075000 [ 695 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-10-29 20:20:04.094000 [ 695 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 20:20:04.094000 [ 695 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-10-29 20:20:04.094000 [ 695 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-10-29 20:20:04.095000 [ 695 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-10-29 20:20:04.111000 [ 695 ] DEBUG : Stdout:12 (cluster.py:145, run_and_check) 2026-10-29 20:20:04.112000 [ 695 ] DEBUG : Command:[docker volume prune -f] (cluster.py:121, run_and_check) 2026-10-29 20:20:04.133000 [ 695 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-10-29 20:20:04.134000 [ 695 ] DEBUG : Volumes pruned: 12 (cluster.py:915, cleanup) =============================== warnings summary =============================== test_reload_client_certificate/test.py: 33 warnings /usr/local/lib/python3.10/dist-packages/kazoo/handlers/utils.py:225: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ----------------- generated report log file: parallel2_0.jsonl ----------------- ============================== slowest durations =============================== 181.05s call test_reload_client_certificate/test.py::test_wrong_cn_cert 41.22s call test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards 40.52s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] 40.37s call test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 37.72s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] 34.88s call test_recovery_replica/test.py::test_update_metadata 33.49s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 33.00s setup test_read_only_table/test.py::test_restart_zookeeper 33.00s call test_recovery_replica/test.py::test_choose_source_replica 31.24s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] 26.66s teardown test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 25.36s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] 24.19s setup test_reloading_storage_configuration/test.py::test_add_disk 23.98s call test_reload_client_certificate/test.py::test_correct_cn_cert 22.91s setup test_replication_credentials/test.py::test_credentials_and_no_credentials 22.74s setup test_recovery_replica/test.py::test_choose_source_replica 22.30s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 22.14s setup test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 22.02s teardown test_s3_style_link/test.py::test_s3_table_functions_line_as_string 21.80s teardown test_s3_imds/test_simple.py::test_credentials_from_metadata 21.72s teardown test_s3_table_functions/test.py::test_s3_table_functions_timeouts 21.66s teardown test_s3_latency_log/test.py::test_latency_log 21.64s teardown test_s3_storage_class/test.py::test_s3_storage_class_right 21.58s teardown test_s3_imds/test_session_token.py::test_credentials_from_metadata 21.35s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] 21.26s setup test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards 21.17s teardown test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] 21.12s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] 21.00s call test_partition/test.py::test_system_detached_parts 20.79s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter 19.84s teardown test_part_uuid/test.py::test_part_uuid 19.70s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system 18.46s setup test_part_uuid/test.py::test_part_uuid 18.37s setup test_replicating_constants/test.py::test_different_versions 18.36s call test_read_only_table/test.py::test_restart_zookeeper 18.20s teardown test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 17.89s call test_s3_latency_log/test.py::test_latency_log 17.79s setup test_s3_latency_log/test.py::test_latency_log 17.47s setup test_reload_client_certificate/test.py::test_correct_cn_cert 17.33s teardown test_reload_client_certificate/test.py::test_wrong_cn_cert 17.24s setup test_s3_low_cardinality_right_border/test.py::test_s3_right_border 17.15s setup test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 17.14s call test_rocksdb_options/test.py::test_invalid_options 16.95s setup test_partition/test.py::test_attach_check_all_parts 16.75s call test_rocksdb_options/test.py::test_table_invalid_options 16.44s setup test_s3_imds/test_simple.py::test_credentials_from_metadata 16.42s call test_rocksdb_options/test.py::test_table_invalid_column_family_options 16.29s call test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 15.18s setup test_s3_table_functions/test.py::test_s3_table_functions 14.85s call test_rocksdb_options/test.py::test_invalid_column_family_options 14.77s call test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards 14.74s setup test_rocksdb_options/test.py::test_invalid_block_based_table_options 14.67s setup test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] 14.42s setup test_s3_imds/test_session_token.py::test_credentials_from_metadata 14.30s setup test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 13.99s setup test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 13.67s setup test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race 13.53s call test_rocksdb_options/test.py::test_invalid_block_based_table_options 13.41s setup test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 13.12s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] 13.05s call test_s3_table_functions/test.py::test_s3_table_functions_timeouts 12.83s teardown test_replication_credentials/test.py::test_same_credentials 12.72s setup test_relative_filepath/test.py::test_filepath 12.55s setup test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 12.44s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] 11.94s call test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 11.18s call test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] 11.02s call test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 10.47s setup test_s3_storage_class/test.py::test_s3_storage_class_right 10.33s call test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race 10.15s call test_reloading_storage_configuration/test.py::test_add_disk 9.96s setup test_s3_style_link/test.py::test_s3_table_functions 9.93s teardown test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards 9.70s call test_reloading_storage_configuration/test.py::test_new_policy_works 9.62s call test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 9.55s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge 9.40s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter 9.39s call test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 8.63s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 8.57s call test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 8.53s call test_reloading_storage_configuration/test.py::test_remove_policy 8.40s call test_reloading_storage_configuration/test.py::test_add_policy 8.35s call test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 8.21s call test_recovery_replica/test.py::test_recovery 8.20s call test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 8.11s teardown test_read_only_table/test.py::test_restart_zookeeper 7.99s call test_reloading_storage_configuration/test.py::test_remove_disk 7.89s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] 7.77s call test_reloading_storage_configuration/test.py::test_update_disk 7.77s call test_reloading_storage_configuration/test.py::test_add_disk_to_policy 7.73s call test_reloading_storage_configuration/test.py::test_add_volume_to_policy 7.64s teardown test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 7.53s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] 7.38s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] 6.76s call test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 6.74s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 6.59s call test_replication_credentials/test.py::test_credentials_and_no_credentials 6.58s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] 6.48s call test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 6.28s call test_s3_low_cardinality_right_border/test.py::test_s3_right_border 6.27s call test_partition/test.py::test_detached_part_dir_exists 6.11s call test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 6.04s teardown test_recovery_replica/test.py::test_update_metadata 5.94s call test_part_uuid/test.py::test_part_uuid 5.90s teardown test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 5.89s call test_replication_credentials/test.py::test_different_credentials 5.81s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] 5.70s call test_partition/test.py::test_make_clone_in_detached 5.60s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] 5.57s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] 5.44s call test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] 5.37s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] 5.34s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] 5.31s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] 5.23s call test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] 5.14s teardown test_partition/test.py::test_system_detached_parts 4.98s call test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 4.87s call test_partition/test.py::test_attach_check_all_parts 4.42s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] 4.36s teardown test_replicating_constants/test.py::test_different_versions 4.02s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] 3.97s call test_partition/test.py::test_partition_simple 3.92s call test_replication_credentials/test.py::test_same_credentials 3.91s call test_replication_credentials/test.py::test_no_credentials 3.81s call test_partition/test.py::test_drop_detached_parts 3.77s teardown test_reloading_storage_configuration/test.py::test_update_disk 3.62s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 3.61s teardown test_relative_filepath/test.py::test_filepath 3.50s teardown test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 3.46s teardown test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 3.46s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 3.45s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] 3.34s teardown test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race 3.16s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 3.11s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 2.96s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 2.90s call test_partition/test.py::test_partition_complex 2.88s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] 2.81s teardown test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 2.76s call test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 2.70s call test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 2.50s teardown test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 2.07s call test_s3_style_link/test.py::test_s3_table_functions 2.06s call test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 1.97s setup test_partition/test.py::test_drop_detached_parts 1.95s call test_s3_imds/test_session_token.py::test_credentials_from_metadata 1.86s call test_s3_style_link/test.py::test_s3_table_functions_line_as_string 1.83s call test_s3_imds/test_simple.py::test_credentials_from_metadata 1.77s setup test_partition/test.py::test_system_detached_parts 1.70s setup test_partition/test.py::test_partition_simple 1.63s teardown test_rocksdb_options/test.py::test_valid_options 1.63s setup test_replication_credentials/test.py::test_different_credentials 1.48s call test_s3_table_functions/test.py::test_s3_table_functions 1.33s call test_rocksdb_options/test.py::test_valid_options 1.29s setup test_partition/test.py::test_partition_complex 1.23s call test_reload_query_masking_rules/test.py::test_reload_query_masking_rules 1.03s call test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 0.97s call test_partition/test.py::test_cannot_attach_active_part 0.90s call test_s3_storage_class/test.py::test_s3_storage_class_right 0.87s call test_range_hashed_dictionary_types/test.py::test_range_hashed_dict 0.87s setup test_partition/test.py::test_cannot_attach_active_part 0.79s call test_relative_filepath/test.py::test_filepath 0.77s setup test_replication_credentials/test.py::test_no_credentials 0.76s setup test_replication_credentials/test.py::test_same_credentials 0.54s teardown test_partition/test.py::test_drop_detached_parts 0.54s teardown test_partition/test.py::test_attach_check_all_parts 0.37s teardown test_partition/test.py::test_cannot_attach_active_part 0.37s call test_replicating_constants/test.py::test_different_versions 0.33s teardown test_partition/test.py::test_partition_simple 0.32s call test_rocksdb_options/test.py::test_valid_column_family_options 0.30s call test_rocksdb_options/test.py::test_table_valid_column_family_options 0.27s call test_rocksdb_options/test.py::test_table_valid_options 0.25s teardown test_partition/test.py::test_partition_complex 0.01s teardown test_replication_credentials/test.py::test_credentials_and_no_credentials 0.01s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk 0.00s teardown test_recovery_replica/test.py::test_choose_source_replica 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_recovery_replica/test.py::test_recovery 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 0.00s teardown test_partition/test.py::test_detached_part_dir_exists 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s teardown test_reload_client_certificate/test.py::test_correct_cn_cert 0.00s teardown test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] 0.00s setup test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] 0.00s teardown test_partition/test.py::test_make_clone_in_detached 0.00s teardown test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s setup test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards 0.00s setup test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s teardown test_s3_low_cardinality_right_border/test.py::test_s3_right_border 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] 0.00s setup test_partition/test.py::test_make_clone_in_detached 0.00s teardown test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s setup test_rocksdb_options/test.py::test_invalid_options 0.00s setup test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] 0.00s setup test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s teardown test_s3_table_functions/test.py::test_s3_table_functions 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] 0.00s setup test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s setup test_reloading_storage_configuration/test.py::test_update_disk 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 0.00s setup test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] 0.00s setup test_recovery_replica/test.py::test_update_metadata 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 0.00s setup test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] 0.00s teardown test_rocksdb_options/test.py::test_invalid_block_based_table_options 0.00s teardown test_s3_style_link/test.py::test_s3_table_functions 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] 0.00s setup test_partition/test.py::test_detached_part_dir_exists 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 0.00s setup test_reload_client_certificate/test.py::test_wrong_cn_cert 0.00s setup test_reloading_storage_configuration/test.py::test_remove_policy 0.00s setup test_s3_style_link/test.py::test_s3_table_functions_line_as_string 0.00s setup test_rocksdb_options/test.py::test_table_valid_options 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] 0.00s setup test_s3_table_functions/test.py::test_s3_table_functions_timeouts 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 0.00s teardown test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] 0.00s setup test_reloading_storage_configuration/test.py::test_remove_disk 0.00s setup test_rocksdb_options/test.py::test_table_invalid_options 0.00s setup test_reloading_storage_configuration/test.py::test_add_policy 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s teardown test_reloading_storage_configuration/test.py::test_add_policy 0.00s setup test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge 0.00s setup test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 0.00s setup test_rocksdb_options/test.py::test_valid_options 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 0.00s teardown test_reloading_storage_configuration/test.py::test_new_policy_works 0.00s setup test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s teardown test_replication_credentials/test.py::test_no_credentials 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 0.00s setup test_rocksdb_options/test.py::test_valid_column_family_options 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] 0.00s setup test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 0.00s setup test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_options 0.00s teardown test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] 0.00s teardown test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s teardown test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s setup test_reloading_storage_configuration/test.py::test_new_policy_works 0.00s teardown test_rocksdb_options/test.py::test_invalid_options 0.00s teardown test_replication_credentials/test.py::test_different_credentials 0.00s teardown test_recovery_replica/test.py::test_recovery 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter 0.00s setup test_rocksdb_options/test.py::test_table_invalid_block_based_table_options 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s teardown test_rocksdb_options/test.py::test_valid_column_family_options 0.00s teardown test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] 0.00s teardown test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] 0.00s teardown test_rocksdb_options/test.py::test_table_valid_options =========================== short test summary info ============================ FAILED test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race PASSED test_partition/test.py::test_attach_check_all_parts PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border PASSED test_partition/test.py::test_cannot_attach_active_part PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key] PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 PASSED test_rocksdb_options/test.py::test_invalid_block_based_table_options PASSED test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_replication_credentials/test.py::test_credentials_and_no_credentials PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids PASSED test_partition/test.py::test_detached_part_dir_exists PASSED test_reloading_storage_configuration/test.py::test_add_disk PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_keys PASSED test_partition/test.py::test_drop_detached_parts PASSED test_replication_credentials/test.py::test_different_credentials PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key] PASSED test_replication_credentials/test.py::test_no_credentials PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces PASSED test_partition/test.py::test_make_clone_in_detached PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] PASSED test_rocksdb_options/test.py::test_invalid_column_family_options PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] PASSED test_replication_credentials/test.py::test_same_credentials PASSED test_partition/test.py::test_partition_complex PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_partition/test.py::test_partition_simple PASSED test_replicated_merge_tree_encryption_codec/test.py::test_same_keys PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config PASSED test_recovery_replica/test.py::test_choose_source_replica PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0] PASSED test_rocksdb_options/test.py::test_invalid_options PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)] PASSED test_reloading_storage_configuration/test.py::test_add_policy PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key] PASSED test_recovery_replica/test.py::test_recovery PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3] PASSED test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)] PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op] PASSED test_partition/test.py::test_system_detached_parts PASSED test_rocksdb_options/test.py::test_table_invalid_block_based_table_options PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op] PASSED test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1] PASSED test_reloading_storage_configuration/test.py::test_new_policy_works PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op] PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op] PASSED test_reloading_storage_configuration/test.py::test_remove_disk PASSED test_range_hashed_dictionary_types/test.py::test_range_hashed_dict PASSED test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options PASSED test_s3_style_link/test.py::test_s3_table_functions PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op] PASSED test_s3_style_link/test.py::test_s3_table_functions_line_as_string PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy PASSED test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op] PASSED test_recovery_replica/test.py::test_update_metadata PASSED test_reload_client_certificate/test.py::test_correct_cn_cert PASSED test_reloading_storage_configuration/test.py::test_remove_policy PASSED test_relative_filepath/test.py::test_filepath PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge PASSED test_rocksdb_options/test.py::test_table_invalid_options PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options PASSED test_rocksdb_options/test.py::test_table_valid_options PASSED test_rocksdb_options/test.py::test_valid_column_family_options PASSED test_rocksdb_options/test.py::test_valid_options PASSED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned PASSED test_reloading_storage_configuration/test.py::test_update_disk PASSED test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter PASSED test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards PASSED test_parallel_replicas_increase_error_count/test.py::test_increase_error_count PASSED test_replicating_constants/test.py::test_different_versions PASSED test_read_only_table/test.py::test_restart_zookeeper PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system PASSED test_s3_storage_class/test.py::test_s3_storage_class_right PASSED test_s3_table_functions/test.py::test_s3_table_functions PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop PASSED test_reload_query_masking_rules/test.py::test_reload_query_masking_rules PASSED test_s3_imds/test_session_token.py::test_credentials_from_metadata PASSED test_s3_table_functions/test.py::test_s3_table_functions_timeouts PASSED test_s3_latency_log/test.py::test_latency_log PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation PASSED test_s3_imds/test_simple.py::test_credentials_from_metadata PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system PASSED test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10] PASSED test_reload_client_certificate/test.py::test_wrong_cn_cert PASSED test_part_uuid/test.py::test_part_uuid PASSED test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3] ============ 1 failed, 99 passed, 33 warnings in 392.23s (0:06:32) ============= Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_iwvsmg --privileged --dns-search='.' --memory=30703247360 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 --report-log=parallel2_0.jsonl --report-log-exclude-logs-on-passed-tests 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_multiple_shards_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_distributed[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_mergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_range-sipHash64(key)]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-key]' 'test_parallel_replicas_custom_key/test.py::test_parallel_replicas_custom_key_replicatedmergetree[test_single_shard_multiple_replicas-custom_key_sampling-sipHash64(key)]' test_parallel_replicas_increase_error_count/test.py::test_increase_error_count 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[100-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[2-1]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-0]' 'test_parallel_replicas_no_replicas/test.py::test_skip_all_replicas[3-1]' test_parallel_replicas_skip_shards/test.py::test_error_on_unavailable_shards test_parallel_replicas_skip_shards/test.py::test_skip_unavailable_shards test_part_uuid/test.py::test_part_uuid test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple test_partition/test.py::test_system_detached_parts test_race_condition_for_replicated_merge_tree/test.py::test_partition_move_drop_race test_range_hashed_dictionary_types/test.py::test_range_hashed_dict test_read_only_table/test.py::test_restart_zookeeper test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_relative_filepath/test.py::test_filepath test_reload_client_certificate/test.py::test_correct_cn_cert test_reload_client_certificate/test.py::test_wrong_cn_cert test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_query_masking_rules/test.py::test_reload_query_masking_rules test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned test_replicating_constants/test.py::test_different_versions test_replication_credentials/test.py::test_credentials_and_no_credentials test_replication_credentials/test.py::test_different_credentials test_replication_credentials/test.py::test_no_credentials test_replication_credentials/test.py::test_same_credentials test_rocksdb_options/test.py::test_invalid_block_based_table_options test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_block_based_table_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_s3_imds/test_session_token.py::test_credentials_from_metadata test_s3_imds/test_simple.py::test_credentials_from_metadata test_s3_latency_log/test.py::test_latency_log test_s3_low_cardinality_right_border/test.py::test_s3_right_border test_s3_low_cardinality_right_border/test.py::test_s3_right_border_2 test_s3_low_cardinality_right_border/test.py::test_s3_right_border_3 test_s3_storage_class/test.py::test_s3_storage_class_right 'test_s3_storage_conf_new_proxy/test.py::test_s3_with_proxy_list[s3]' test_s3_style_link/test.py::test_s3_table_functions test_s3_style_link/test.py::test_s3_table_functions_line_as_string test_s3_table_functions/test.py::test_s3_table_functions test_s3_table_functions/test.py::test_s3_table_functions_timeouts 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[None-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_after_op-zero_copy_lock_zk_fail_before_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_after_op]' 'test_s3_zero_copy_replication/test.py::test_move_shared_zero_copy_lock_fail[zero_copy_unlock_zk_fail_before_op-zero_copy_lock_zk_fail_before_op]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_concurrent_merge test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_drop_detached_system test_s3_zero_copy_replication/test.py::test_s3_zero_copy_keeps_data_after_mutation 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_replication[s3]' test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_alter test_s3_zero_copy_replication/test.py::test_s3_zero_copy_unfreeze_system 'test_s3_zero_copy_replication/test.py::test_s3_zero_copy_with_ttl_delete[False-10]' -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 ' returned non-zero exit status 1.